Augmented reality usage for prototyping speed up

ŠŤASTNÝ, J., PROCHÁZKA, D., KOUBEK, T., LANDA, J.: Augmented reality usage for prototyping speed up. Acta univ. agric. et silvic. Mendel. Brun., 2011, LIX, No. 2, pp. 353–360 The integral part of production process in many companies is prototyping. Although, these companies commonly have high quality visualization tools (large screen projections, virtual reality), prototyping was never abandoned. There is a number of reasons. The most important is the possibility of model observation from any angle without any physical constraints and its haptic feedback. The interactivity of model adjustments is important as well. The direct work with the model allows the designers to focus on the creative process more than work with a computer. There is still a problem with a diffi cult adjustability of the model. More signifi cant changes demand completely new prototype or at least longer time for its realization. The fi rst part of the article describes our approach for solution of this problem by means of Augmented Reality. The merging of the real world model and digital objects allows streamline the work with the model and speed up the whole production phase signifi cantly. The main advantage of augmented reality is the possibility of direct manipulation with the scene using a portable digital camera. Also adding digital objects into the scene could be done using identifi cation markers placed on the surface of the model. Therefore it is not necessary to work with special input devices and lose the contact with the real world model. Adjustments are done directly on the model. The key problem of outlined solution is the ability of identifi cation of an object within the camera picture and its replacement with the digital object. The second part of the article is focused especially on the identifi cation of exact position and orientation of the marker within the picture. The identifi cation marker is generalized into the triple of points which represents a general plane in space. There is discussed the space identifi cation of these points and the description of representation of their position and orientation be means of transformation matrix. This matrix is used for rendering of the graphical objects (e. g. in OpenGL and Direct3D). augmented reality, prototyping, pose estimation, transformation matrix European manufacturers face strong competitive pressure which makes them speed up the development and production process constantly. One of the reasons is the market globalization. In many countries extremely low production costs are possible. The prize is o en a violation of ethical and safety principles that is unacceptable in most European countries. One of the possibilities how to fi ght with such a kind of production is to maintain a technological lead. Therefore it is necessary to focus on methods which increase the development and production process effi ciency. An integral part of development of many products is their design. It is well known that the process includes initial design phase in a form of sketches which is usually followed by an electronic visualization phase of potential solutions (3D models development). The following step is usually prototyping. The gist of creation of even non-functional prototype is to obtain a clear image about its design and practically test its ergonomics. (Is it comfortable to hold the device? What is the fi eld-of-view from the rear window of the car?). 354 J. Šťastný, D. Procházka, T. Koubek, J. Landa Promising methods for streamlining the last mentioned phase are the virtual and augmented reality. The experiments with deployment of these visualization techniques have been ongoing for many years. Although it is possible to present a lot of partial successes with their deployment, their usage is still not common. In the following part we discuss possible reasons of their diffi cult integration to the production process. Based on this analysis, we propose a new method for improvement of the prototyping process using augmented reality and we describe a solution of the key problem in detail – pose identifi cation of an object. METHODS AND RESOURCES Various advanced visualization techniques such as large and stereoscopic projections are a common part of the second design phase – development of 3D models or as a part of a decision-making process, presentation or marketing. In prototyping phase the virtual reality technology is a well established tool almost exclusively for testing of functional properties. We can also mention the signifi cance of CAVE (stereoscopic projection surrounding the observer) which can be used to simulate the interior of e.g. car or plane. In these applications CAVE can exceed the prototype creation – there are no limitations in size and it is possible to simulate the functionality of the interior. The typical example is an article by Dmitriev et al., 2004 where advanced visualization of a car interior is presented. Not entirely resolved problem is a complete replacement of the real prototype with a virtual one. Such a replacement will be possible a er fulfi lling the following criteria: The model has excellent visual quality and it is possible to manipulate with it naturally (including the haptic feedback). The fi rst requirement can be fulfi lled in some detail. This aspect is discussed in Ong-Nee, 2004, p. 15–42 and Choi-Cheung, 2008. Much more significant problem is the natural manipulation with the object which includes the haptic feedback (see OngNee, 2004, p. 43–64). Prototyping tools which use various forms of stereoscopic projection usually offer a limited functionality in comparison to the reality. Current technologies allow haptic feedback simulation (see Kortum, 2008), however the naturalness of these methods is o en problematic (e.g. Johnson et al., 2005). During the product design phase it is crucial for a designer not to be limited by any hardware and so ware constraints. These constraints could have signifi cant infl uence on the quality of his work. The poor result of various virtual tools does not have to be caused by limited functionality but more likely by psychological barrier of the designer. These cases are relatively common in many virtual reality applications (Carroll, 2005). Because of these reasons the physical model was never completely replaced. The question is whether the replacement is all possible. At least until a completely new generation of visualization techniques will be developed. The hope in this area comes with so-called physical holograms (Iwamoto et al., 2008). An interesting contribution for the area mentioned is the augmented reality. It represents a certain form of compromise. A physical model is not completely replaced but its main disadvantage – diffi cult extendability – is suppressed. Virtual objects could immediately extend the real model thus they increase the interactivity of the adjustments. There is almost no experience with such an augment reality deployment. Current industrial augmented reality applications are focused on the construction and maintenance of complex systems only. Details can be found in Ong-Lee, 2004, p. 237–383, Bottecchia et al., 2010 and Bimber-Ramesh 2005. The reason is particularly in the hardware limitations. Immersive augmented reality is almost exclusively realized be means of the extensive hardware. These solutions are acceptable for manual activities but hardly acceptable for creative process. Current applications of virtual and augmented reality have the following common features: The biggest problems are caused by using special hardware and so ware especially lack of the hardware maturity and psychological aspects with its use. In both cases the result is dissatisfaction with the deployment of the AR technology. In order to eliminate this problem required approach must not limit the user. In that case even the psychological barrier – to adopt the solution – will cease to exist. The related problem lies in no mainstream solutions for discussed applications. Solutions must be usually implemented directly for the customer. It leads to higher costs. Therefore the companies are very careful with the deployment of these technologies. The problem will cease to exist a er general solution is developed. Similar problems were e.g. in the beginnings of common stereoscopic projection. Augmented reality applications are not necessarily connected with a special hardware. Also the basic principle of augmenting the scene is general enough to be implemented as a framework independent on a given problem. There are already experiments with such frameworks. The example could be the ARToolKit framework (described in Kato, 1999). The basic application functionality lies in the localization of a marker in a given image and in pairing the marker with a specifi c virtual object. This step is called a registration. The registration is usually done by using an optical sensor – e.g. a digital camera. The issue of optical tracking can be divided into several sub-problems: • Localization of the marker – the marker can be e.g. a Ping-Pong ball or a square shaped image attached to a surface. • Computation of the marker position and orientation – recalculating marker position is a very important part of registration. Resolving this subproblem will ensure the right orientation of an augmented object. Augmented reality usage for prototyping speed up 355 General solution of the fi rst sub-problem is presented in Fig. 1. In the fi rst step an image which contains the marker is obtained. Localization of the marker could be done in several steps – image thresholding, connected components labelling and fi nally identifi cation of the marker vertices. Then the localized marker must be projected to the camera plane and compared with a template stored in the application. Before the object localization itself the image is usually preprocessed. That could reduce redundant information. The typically used method in preprocessing is thresholding. The scene is converted into a gray-scale and a threshold value must be determined. The threshold value is set according to the nature of the image. The pixel which has its value under the threshold is set to black and the pixel with the value above the threshold is set to white. This process is described in detail in Jähne, 2005. Thresholding the captured image allows to fi lter out a marker background. The result of this operation is the thresholded black and white image which is further processed. The principle of connected components labelling lies in the comparison of neighbour pixel values. Each pixel is tested and in case of having the same value as a current component it is designated as a part of this connected component. Otherwise the pixel is labelled as the fi rst pixel of a new component. More information about connected components is written in Acharya – Ray, 2005 or Di Stefano – Bulgarelli, 1999. A er this labelling phase an image with highlighted individual components is obtained. These parts represent our potential markers. The next step is localization of particular points of the marker. If the markers are e.g. squares it is the best to fi nd their corners. The basic algorithm for fi nding corners is based on determining cornerness value for every pixel. This value determines the probability of every pixel being the corner pixel. From these points the cornerness map is created and the local maximums are found. There are diff erent algorithms for fi nding out the cornerness value – e.g. Harris operator. The principle of the operator lies in comparing pixel values in small sectors of image. This algorithm is described e.g. in Mohanna – Mohktarian, 2001 or Rockett, 2003. A er performing all these operations the corners of the marker are available. For proper marker registration it is now necessary to create a transformation which allows us to project the object given by these corners from the camera perspective to its original shape. Then the comparison with the template could be done and in case of the positive result a virtual object could be inserted. The exact method of the projection must be defi ned by virtue the required functionality of the application. Our proposed method is described in the following section.